Automated transition classification for  binge watching of content

ABSTRACT

Novel techniques are described for automated transition classification for binge watching of content. For example, a number of frame images is extracted from a candidate segment time window of content. The frame images can automatically be classified by a trained machine learning model into segment and non-segment classifications, and the classification results can be represented by a two-dimensional (2D) image. The 2D image can be run through a multi-level convolutional conversion to output a set of output images, and a serialized representation of the output images can be run through a trained computational neural network to generate a transition array, from which a candidate transition time can be derived (indicating a precise time at which the content transitions to the classified segment).

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/109,755, filed Aug. 23, 2018, the full disclosure of which isincorporated herein by reference.

FIELD

This invention relates generally to automated image classification, and,more particularly, to automated transition classification for bingewatching of content.

BACKGROUND

In recent years, on-demand content, including television programming andother episodic media content offered by over-the-top (OTT) and othermedia sources has become increasingly available to consumers. In thewake of this availability, consumers are increasingly watching multipleepisodes of such episodic programming in rapid succession; a type ofmedia interaction referred to as binge watching. When binge watchingprogramming, consumers often desire to watch only the portion of theprogramming providing new content for the present episode (referred toherein as the “subject” portion or portions of the program), and may befrustrated by having to watch other portions of the programming, such asopening credits, ending credits, scenes from the last episode providedat the start of a present episode, scenes from the next episode providedat the end of a present episode, etc. (referred to herein as the“non-subject” portion or portions of the program).

Content playback interfaces (e.g., television receivers, and the like,providing OTT streaming episodic media) do not tend to provide anautomated approach to finding transition points in programming betweenthe subject and non-subject portions of a program. For example, contentplayback interfaces providing OTT media tend not to include technologyto automatically identify the transition time at which end credits beginrolling. Identifying such a transition time can appreciably improve aconsumer's binge watching experience, for example, by facilitating theconsumer being able to jump to the next episode as soon as the currentepisode ends (i.e., without having to wait for the end credits to roll,or without requiring other navigation, such as fast-forwarding, ornavigating out to a menu screen).

Some current approaches to identifying transition times, such as an endcredit start time, use crowdsourcing. With such approaches, one or morehuman users can manually identify the transition time and can submit theidentified time to a system. The system can then record the time as thetransition time (or can process multiple received candidate times, anduse a statistical or other technique to determine an appropriatetransition time). Such approaches can be limited in a number of ways.For example, implementations relying on multiple sources of feedback maynot obtain sufficient amounts of feedback quickly enough to be useful inmany instances (e.g., for new or unpopular television programs).Further, the accuracy of such approaches can be subject to human error(e.g., in identifying timing, in data entry, etc.), to fuzziness andunpredictability in the turnout and/or accuracy of human users, to waittimes for crowd responses, etc. Some such approaches can also depend onoffering sufficient incentives to obtain accurate, timely results.

BRIEF SUMMARY

Among other things, embodiments provide novel systems and methods forautomated transition classification for binge watching of content. Forexample, a number of frame images is extracted from a candidate segmenttime window (e.g., a time window during which content is likely totransition between not being part of a type of segment to being part ofthe type of segment, such as transitioning to an ending creditssegment). The frame images can automatically be classified by a trainedmachine learning model into segment and non-segment classifications, andthe classification results can be represented by a two-dimensional (2D)image. The 2D image can be run through a multi-level convolutionalconversion to output a set of output images, and a serializedrepresentation of the output images can be run through a trainedcomputational neural network to generate a transition array, from whicha candidate transition time can be derived (indicating a precise time atwhich the content transitions to the classified segment).

According to one set of embodiments, a method for automated transitionclassification of media programming content is provided. The methodincludes: extracting N frame images at a sampling rate over anidentified candidate segment time window of the media programmingcontent; associating each of the N frame images automatically with arespective element of a 1-by-N input classification array, each elementof the input classification array comprising an element value indicatinga result of automatically classifying the associated frame image intoone of a plurality of predetermined classifications, the plurality ofpredetermined classifications comprising at least a first classificationcorresponding to the frame image being part of a classified segment, anda second classification corresponding to the frame image not being partof the classified segment; converting the input classification arrayautomatically into a two-dimensional input classification image havingan input resolution corresponding to N, and a color depth correspondingto the plurality of predetermined classifications, such that N inputpixel values of the input classification image correspond to the Nelement values of the classification array; computing a plurality ofoutput classification images automatically as a function of running theinput classification image automatically through a multi-levelconvolutional conversion, the plurality of output classification imageshaving a total of M output pixel values, where M is greater than N;serializing the M output pixel values automatically to a 1-by-M outputclassification array, such that M element values of the outputclassification array correspond to the M output pixel values of theplurality of output classification images; computing a 1-by-N transitionarray automatically as a function of running the output classificationarray through an computational neural network trained to compute each ofN element values of the transition array as a function of the M elementvalues of the output classification array; and outputting a candidatetransition time as a function of the transition array, the candidatetransition time corresponding to a temporal location at which the mediaprogramming content transitions between not being part of the classifiedsegment and being part of the classified segment.

According to another set of embodiments, a system is provided forautomated transition classification of media programming content. Thesystem includes: an image extraction controller comprising a video datainput and a frame image output, the frame image output generated byextracting N frame images from the video data input at a sampling rateover an identified candidate segment time window; an imageclassification controller having a first machine learning model storedthereon, and coupled with the image extraction controller to output a1-by-N input classification array, the input classification arraycomputed as a function of applying the first machine learning model tothe N frame images by associating each of the N frame images with arespective element of the input classification array, such that eachelement of the input classification array comprises an element valueindicating one of a plurality of predetermined classificationscomprising a first classification corresponding to the frame image beingpart of a classified segment, and a second classification correspondingto the frame image not being part of the classified segment; and atransition frame identification controller having a second machinelearning model stored thereon, and coupled with the image classificationcontroller to output a candidate transition time according to a 1-by-Ntransition array, the candidate transition time corresponding to atemporal location at which the video data transitions between not beingpart of the classified segment and being part of the classified segment.The transition array is computed as a function of applying the secondmachine learning model to the input classification array by: convertingthe input classification array into a two-dimensional inputclassification image having an input resolution corresponding to N and acolor depth corresponding to the plurality of predeterminedclassifications, such that N input pixel values of the inputclassification image correspond to the N element values of theclassification array; computing a plurality of output classificationimages automatically as a function of running the input classificationimage automatically through a multi-level convolutional conversion, theplurality of output classification images having a total of M outputpixel values, where M is greater than N; and computing the 1-by-Ntransition array as a function of applying an computational neuralnetwork to the M output pixel values. Some such embodiments furtherinclude a binge viewing controller to generate a playback navigationcontrol, responsive to a user interface command, to skip at least aportion of the classified segment in accordance with the candidatetransition time.

According to another set of embodiments, a non-transient,machine-readable medium is provided. The medium is coupled with one ormore processors and has, stored thereon, instructions, which, whenexecuted, cause the one or more processors to perform steps comprising:extracting N frame images at a sampling rate over an identifiedcandidate segment time window of the media programming content;associating each of the N frame images with a respective element of a1-by-N input classification array, each element of the inputclassification array comprising an element value indicating a result ofautomatically classifying the associated frame image into one of aplurality of predetermined classifications, the plurality ofpredetermined classifications comprising at least a first classificationcorresponding to the frame image being part of a classified segment, anda second classification corresponding to the frame image not being partof the classified segment; converting the input classification arrayinto a two-dimensional input classification image having an inputresolution corresponding to N, and a color depth corresponding to theplurality of predetermined classifications, such that N input pixelvalues of the input classification image correspond to the N elementvalues of the classification array; computing a plurality of outputclassification images as a function of running the input classificationimage automatically through a multi-level convolutional conversion, theplurality of output classification images having a total of M outputpixel values, where M is greater than N; serializing the M output pixelvalues to a 1-by-M output classification array, such that M elementvalues of the output classification array correspond to the M outputpixel values of the plurality of output classification images; computinga 1-by-N transition array as a function of running the outputclassification array through an computational neural network trained tocompute each of N element values of the transition array as a functionof the M element values of the output classification array; andoutputting a candidate transition time as a function of the transitionarray, the transition time corresponding to a temporal location at whichthe media programming content transitions between not being part of theclassified segment and being part of the classified segment.

This summary is not intended to identify key or essential features ofthe claimed subject matter, nor is it intended to be used in isolationto determine the scope of the claimed subject matter. The subject mattershould be understood by reference to appropriate portions of the entirespecification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and embodiments, will becomemore apparent upon referring to the following specification, claims, andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIG. 1 shows an illustrative satellite television distribution system ascontext for various embodiments described herein;

FIG. 2 shows a block diagram of an illustrative system for automatedtransition classification of media programming content;

FIG. 3 shows a data flow diagram for an illustrative implementation ofembodiments of the image classification controller and the transitionframe identification controller of FIG. 2;

FIG. 4 provides a schematic illustration of one embodiment of a computersystem that can implement various system components and/or performvarious steps of methods provided by various embodiments describedherein; and

FIG. 5 shows a flow diagram of an illustrative method, according tovarious embodiments.

In the appended figures, similar components and/or features may have thesame reference label. Further, various components of the same type maybe distinguished by following the reference label by a second label(e.g., a lower-case letter) that distinguishes among the similarcomponents. If only the first reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same first reference label irrespective of thesecond reference label.

DETAILED DESCRIPTION

Embodiments of the disclosed technology will become clearer whenreviewed in connection with the description of the figures herein below.In the following description, numerous specific details are set forth toprovide a thorough understanding of the present invention. However, onehaving ordinary skill in the art should recognize that the invention maybe practiced without these specific details. In some instances,circuits, structures, and techniques have not been shown in detail toavoid obscuring the present invention.

FIG. 1 shows an illustrative satellite television distribution system100 as context for various embodiments described herein. Satellitetelevision distribution system 100 may include: television serviceprovider system 110, satellite transmitter equipment 120, satellites130, satellite dish 140, television receiver 150, and display device160. Alternate embodiments of satellite television distribution system100 may include fewer or greater numbers of components. While only onesatellite dish 140, television receiver 150, and display device 160(collectively referred to as “user equipment”) are illustrated, itshould be understood that multiple (e.g., tens, thousands, millions)instances of user equipment may receive television signals fromtelevision service provider system 110 via satellites 130. Further,while embodiments are described in particular context of a satellitetelevision distribution system 100, techniques described herein can alsobe implemented in context of other television and media distributionarchitectures and infrastructures, such as cable television distributionnetworks.

The user equipment is shown disposed at a subscriber premises 145. Thesubscriber premises 145 can be a subscriber's home, office, or anysuitable location associated with a particular subscriber at which tolocate the subscriber's satellite dish 140 and television receiver 150.As one example, the satellite dish 140 is mounted to an exteriorlocation (e.g., wall, roof, etc.) of the subscriber's home, and cablesrun from the satellite dish 140 to the television receiver 150, which islocated inside the subscriber's home; and one or more display devices160 (e.g., televisions) is coupled with the television receiver 150 andlocated in the subscriber's home.

Television service provider system 110 and satellite transmitterequipment 120 may be operated by a television service provider. Atelevision service provider may distribute television channels,on-demand programming, programming information, and/or othercontent/services to users. Television service provider system 110 mayreceive feeds of one or more television channels from various sources.Such television channels may include multiple television channels thatcontain at least some of the same content (e.g., network affiliates). Todistribute television channels for presentation to users, feeds of thetelevision channels may be relayed to user equipment via multipletelevision distribution satellites. Each satellite may relay multipletransponder streams. Satellite transmitter equipment 120 (120-1, 120-2)may be used to transmit a feed of one or more television channels fromtelevision service provider system 110 to one or more satellites 130.While a single television service provider system 110 and satellitetransmitter equipment 120 are illustrated as part of satellitetelevision distribution system 100, it should be understood thatmultiple instances of transmitter equipment may be used, possiblyscattered geographically, to communicate with satellites 130. Suchmultiple instances of satellite transmitting equipment 120 maycommunicate with the same or with different satellites 130. Differenttelevision channels may be transmitted to satellites 130 from differentinstances of transmitting equipment 120. For instance, a differentsatellite dish of satellite transmitter equipment 120 may be used forcommunication with satellites 130 in different orbital slots.

Satellites 130 may be configured to receive signals, such as streams oftelevision channels, from one or more satellite uplinks such assatellite transmitter equipment 120. Satellites 130 may relay receivedsignals from satellite transmitter equipment 120 (and/or other satellitetransmitter equipment) to multiple instances of user equipment viatransponder streams. Different frequencies may be used for uplinksignals 170 from transponder streams 180. Satellites 130 may be ingeosynchronous orbit. Each of the transponder streams transmitted bysatellites 130 may contain multiple television channels transmitted aspacketized data. For example, a single transponder stream may be aserial digital packet stream containing multiple television channels.Therefore, packets for multiple television channels may be interspersed.

Multiple satellites 130 may be used to relay television channels fromtelevision service provider system 110 to satellite dish 140. Differenttelevision channels may be carried using different satellites 130.Different television channels may also be carried using differenttransponders of the same satellite 130; thus, such television channelsmay be transmitted at different frequencies and/or different frequencyranges. As an example, a first and second television channel may berelayed via a first transponder of satellite 130-1. A third, fourth, andfifth television channel may be relayed via a different satellite or adifferent transponder of the same satellite relaying a transponderstream at a different frequency. A transponder stream transmitted by aparticular transponder of a particular satellite may include a finitenumber of television channels, such as seven. Accordingly, if manytelevision channels are to be made available for viewing and recording,multiple transponder streams may be necessary to transmit all of thetelevision channels to the instances of user equipment. Each transponderstream may be able to carry a finite amount of data. As such, the numberof television channels that can be included in a particular transponderstream may be at least partially dependent on the resolution of thevideo of the television channel. For example, a transponder stream maybe able to carry seven or eight television channels at a highresolution, but may be able to carry dozens, fifty, a hundred, twohundred, or some other number of television channels at reducedresolutions.

Satellite dish 140 may be a piece of user equipment that is used toreceive transponder streams from one or more satellites, such assatellites 130. Satellite dish 140 may be provided to a subscriber foruse on a subscription basis to receive television channels provided bythe television service provider system 110, satellite transmitterequipment 120, and/or satellites 130. Satellite dish 140, which mayinclude one or more low noise blocks (LNBs), may be configured toreceive transponder streams from multiple satellites and/or multipletransponders of the same satellite. Satellite dish 140 may be configuredto receive television channels via transponder streams on multiplefrequencies. Based on the characteristics of television receiver 150and/or satellite dish 140, it may only be possible to capturetransponder streams from a limited number of transponders concurrently.For example, a tuner of television receiver 150 may only be able to tuneto a single transponder stream from a transponder of a single satelliteat a given time. The tuner can then be re-tuned to another transponderof the same or a different satellite. A television receiver 150 havingmultiple tuners may allow for multiple transponder streams to bereceived at the same time.

In communication with satellite dish 140 may be one or more televisionreceivers 150. Television receivers 150 may be configured to decodesignals received from satellites 130 via satellite dish 140 for outputand presentation via a display device, such as display device 160. Atelevision receiver may be incorporated as part of a television or maybe part of a separate device, commonly referred to as a set-top box(STB). Television receiver 150 may decode signals received via satellitedish 140 and provide an output to display device 160. A televisionreceiver is defined to include set-top boxes (STBs) and also circuitryhaving similar functionality that may be incorporated with anotherdevice. For instance, circuitry similar to that of a television receivermay be incorporated as part of a television, and/or such circuitry canbe implemented in multiple appliances. While FIG. 1 illustrates anembodiment of television receiver 150 as separate from display device160, it should be understood that, in other embodiments, similarfunctions may be performed by a television receiver integrated withdisplay device 160.

As shown, the television receiver 150 can be implemented as one or moreappliances 152. For example, the television receiver 150 can include aSTB and an over-the-top (OTT) appliance, a master STB in communicationwith distributed slave STBs, etc. Though not shown, some implementationsof the television receiver 150 include a modem, or the like. Forexample, the illustrated satellite network can be used to provide bothtelevisions services and other communications services, such as Internetservices. In such implementations, the network 190 can be implemented bythe satellite communications network. For example, the output of themodem can be coupled (e.g., via a splitter) to both television receiver150 components (e.g., a STB) and a local area network (LAN) router, orthe like. The television receiver 150 can also include a binge controlsystem 211, which is described in more detail below.

Display device 160 may be used to present video and/or audio decoded andoutput by television receiver 150. Television receiver 150 may alsooutput a display of one or more interfaces to display device 160, suchas an electronic programming guide (EPG). In many embodiments, displaydevice 160 is a television. Display device 160 may also be a monitor,computer, or some other device configured to display video and,possibly, play audio.

Uplink signal 170-1 represents a signal between satellite transmitterequipment 120 and satellite 130-1. Uplink signal 170-2 represents asignal between satellite transmitter equipment 120 and satellite 130-2.Each of uplink signals 170 may contain streams of one or more differenttelevision channels. For example, uplink signal 170-1 may contain afirst group of television channels, while uplink signal 170-2 contains asecond group of television channels. Each of these television channelsmay be scrambled such that unauthorized persons are prevented fromaccessing the television channels.

Transponder stream 180-1 represents a transponder stream signal betweensatellite 130-1 and satellite dish 140. Transponder stream 180-2represents a transponder stream signal between satellite 130-2 andsatellite dish 140. Each of transponder streams 180 may contain one ormore different television channels, which may be at least partiallyscrambled. For example, transponder stream 180-1 may be a firsttransponder stream containing a first group of television channels,while transponder stream 180-2 may be a second transponder streamcontaining a different group of television channels. When a televisionchannel is received as part of a transponder stream and is decoded andoutput to display device 160 (rather than first storing the televisionchannel to a storage medium as part of DVR functionality then lateroutputting the television channel from the storage medium), thetelevision channel may be considered to be viewed “live.”

FIG. 1 illustrates transponder stream 180-1 and transponder stream 180-2being received by satellite dish 140 and distributed to televisionreceiver 150. For a first group of television channels, satellite dish140 may receive transponder stream 180-1 and for a second group ofchannels, transponder stream 180-2 may be received. Television receiver150 may decode the received transponder streams. As such, depending onwhich television channels are desired to be presented or stored, varioustransponder streams from various satellites may be received,descrambled, and decoded by television receiver 150.

Network 190 may serve as a secondary communication channel betweentelevision service provider system 110 and television receiver 150.However, in many instances, television receiver 150 may be disconnectedfrom network 190 (for reasons such as because television receiver 150 isnot configured to connect to network 190 or a subscriber does not desireor cannot connect to network 190). As such, the connection betweennetwork 190 and television receiver 150 is represented by a dotted line.Via such a secondary communication channel, bidirectional exchange ofdata may occur. As such, data may be transmitted to television serviceprovider system 110 from television receiver 150 via network 190. Datamay also be transmitted from television service provider system 110 totelevision receiver 150 via network 190. Network 190 may be theInternet. While audio and video services may be provided to televisionreceiver 150 via satellites 130, feedback from television receiver 150to television service provider system 110 may be transmitted via network190.

FIG. 2 shows a block diagram of an illustrative system 200 for automatedtransition classification of media programming content. The system 200can be an implementation of the binge control system 155 of FIG. 1. Forexample, the system 200 can be implemented as part of a televisionreceiver, or in an appliance that is coupled with a television receiver.In other embodiments, some or all of the system 200 is implemented aspart of the television service provider system 110. For example, mediaprogramming content is automatically processed to identify transitiontimes for one or more types of classified segment, and the transitiontimes are stored by a component of the television service providersystem 110. In such embodiments, the transition times can be distributedto binge control systems 155 of subscriber-side television receivers asneeded, or in any other suitable manner.

As illustrated, embodiments of the system 200 can include an imageextraction controller 210, an image classification controller 220, and atransition frame identification controller 240. Some embodiments canfurther include a binge viewing controller 260, as described below.Embodiments of the image extraction controller 210 can include a videodata input 205 and a frame image output 215. The frame image output 215can be generated by extracting N frame images from the video data input205 at a sampling rate over an identified candidate segment time window.In some implementations, the identified candidate segment time window isJ minutes long; and the sampling rate is K samples per second, such thatN=J*K*60. As an example, the identified candidate segment time window isten minutes long; and the sampling rate is one sample per second, suchthat N is 600 frame images. In some implementations, the sampling rateis irregular. For example, the sampling rate may be set to extract 600samples over a ten-minute window, but the time from one sample to thenext may or may not by one second in every instance.

Embodiments of the image classification controller 220 can have a firstmachine learning model, a machine-implemented auto-classification model230, stored thereon. The auto-classification model 230 can beimplemented as any suitable machine learning model trained (according toclassification training data 235) to discriminate between predeterminedclassifications of frame images, as described herein. In someembodiments, the auto-classification model 230 is trained as a functionof the classification training data 235 having a large number oftraining frame images associated with the classified segment, and alarge number of training frame images associated with a segment that isnot the classified segment. For example, a large number of credit frames(e.g., frames that are part of the end credits of one or more programs)and non-credit frames can be used to train the machine learning model torecognize differences between credit and non-credit frames. In some suchimplementations, principle component analysis, or the like, is used by amachine to automatically identify which parameters are most useful indiscriminating between credit and non-credit frames (or any othersuitable classified segment of interest). In some implementations, theauto-classification model 230 can be trained to classify imagesaccording to only one, or more than one, set of predeterminedclassifications corresponding to only one, or more than one type ofclassified segment (e.g., ending credits, opening credits, etc.). Forexample, implementations can be trained to identify frames that are partof ending credits of programs, opening credits of programs, recaps ofepisodes (e.g., a segment at the beginning of an episode that includesclips from one or more previous episodes), previews of episodes (e.g., asegment at the end of an episode that includes clips from one or moreupcoming episodes), supplemental content (e.g., an outtake segment atthe end of an episode), etc.

The image classification controller 220 can be coupled with the imageextraction controller 210 to output a one-dimensional inputclassification array 225 of N elements (i.e., a 1-by-N inputclassification array). The input classification array 225 can becomputed as a function of applying the auto-classification model 230 tothe frame image output 215 (the N frame images) by associating each ofthe N frame images with a respective element of the input classificationarray 225. Each element of the input classification array 225 has anelement value indicating one of multiple predetermined classifications.In some implementations, each element of the input classification arrayhas either a first element value or a second element value (e.g., ‘1’ or‘0’), the first element value indicating that the associated frame imageis part of the classified segment, and the second element valueindicating that the associated frame image is not part of the classifiedsegment. For example, a ‘1’ can indicate that the frame image isclassified as an credit frame, and a ‘0’ can indicate that the frameimage is classified as a non-credit frame. In other implementations,more predetermined classifications can be used.

Embodiments of the transition frame identification controller 240 canhave a second machine learning model, a machine-implementedauto-identification model 250, stored thereon. The auto-identificationmodel 250 can include a multi-level convolutional conversion and acomputational neural network, which are coupled with the imageclassification controller 220 to output transition identification data245 (e.g., a candidate transition time) according to the inputclassification array 225. As described herein, the transition frameidentification controller 240 can convert the input classification array225 into a two-dimensional input classification image, run the inputclassification image through the multi-level convolutional conversion togenerate a number of output classification images, and run the outputclassification images through the computational neural network togenerate the transition identification data 245. For example, thetransition identification data 245 can include a computed candidatetransition time that corresponds to a temporal location at which thevideo data input 205 transitions between not being part of theclassified segment and being part of the classified segment (e.g., thetime at which the ending credits begin, the time at which the openingcredits end, or the like).

FIG. 3 shows a data flow diagram 300 for an illustrative implementationof embodiments of the image classification controller 220 and thetransition frame identification controller 240 of FIG. 2. Embodimentscan begin with a classification model output 310 that indicates a set ofimage frame classifications generated by the image classificationcontroller 220 according to the auto-classification model 230. Theillustrated classification model output 310 is represented as a graphhaving the image frames extracted by the image extraction controller 210along the x-axis, and three classification levels on the y-axis: ‘0’indicates classification as a non-credit frame, ‘1’ indicatesclassification as a credit frame with credits overlaid on otherprogramming in the background, and ‘2’ indicates classification as acredit frame with credits overlaid on a single-color (e.g., black)background. For example, the graph may include N values, each being a‘0’, ‘1’, or ‘2’. At a high level, the classification model output 310appears to indicate that the credits begin to roll about halfway intothe candidate time window, while the program continues to play in thebackground; and, shortly thereafter, the credits continue to roll with ablack background. However, as illustrated, the auto-classification model230 may generate a number of incorrect classifications. For example,over the first half of the candidate time window (e.g., the last tenminutes of a program from which the frame images were extracted), theclassification model output 310 indicates a number of ‘1’ and ‘2’results, corresponding to the auto-classification model 230automatically classifying image frames as credit frames prior to whatappears to be the actual start of the credits. Indeed, if theauto-classification model 230 were perfect, there may be little or noneed for the auto-identification model 250.

The classification model output 310 can be mapped to the inputclassification array 225, as shown. For example, the N results shown inthe classification model output 310 can be mapped sequentially tocorresponding elements of the input classification array 225. In theillustrative case, each element of the input classification array 225can be a ‘0’, ‘1’, or ‘2’; corresponding to a point in the graph of theclassification model output 310. As described above, the inputclassification array 225 can be output from the auto-classificationmodel 230 to the auto-identification model 250.

Embodiments can convert the input classification array 225 into atwo-dimensional input classification image 320. The input classificationimage 320 can have an input resolution corresponding to N, and a colordepth corresponding to the predetermined classifications. For example, Ninput pixel values of the input classification image 320 can correspondto the N element values of the input classification array 225. In someimplementations, converting the input classification array 225 into thetwo-dimensional input classification image 320 includes mapping each ofthe N element values of the input classification array 225 to arespective one of the N input pixel values of the input classificationimage 320, such that the input classification image has a resolution ofB pixels by C pixels, where B*C=N. As an example, 600 extracted frameimages result in 600 classifications, which result in 600 element valuesin the input classification array 225; and the 600 element values in theinput classification array 225 can be converted into a two-dimensionalimage of 24-by-25 pixels (24*25=600 pixels). Each of the 600 pixels ofthe input classification image 320 can be colored according to itscorresponding element of the input classification array 225. In oneimplementation, each pixel is one of three colors (e.g., black, grey, orwhite) corresponding to the three possible classification values, ‘0’,‘1’, and ‘2’ in the illustrated case. In another implementation, eachpixel is one of two colors (e.g., black or white) corresponding togroupings of the three possible classification values (e.g., ‘0’ isencoded as black, corresponding to no credit frames; and both ‘1’ and‘2’ are encoded as white, both corresponding to different types ofcredit frames).

As described above, embodiments of the transition frame identificationcontroller 240 can compute output classification images 335 as afunction of running the input classification image 320 through amulti-level convolutional conversion 330. The illustrated multi-levelconvolutional conversion 330 includes three levels. In a first level,multiple convolutional filters are applied to the input classificationimage 320 to generate a first set of output images 335 a. For example,each convolutional filter effectively enhances some characteristic ofthe input classification image 320 to generate a respective one of thefirst set of output images 335 a. Each of the first set of output images335 a has lower resolution than the resolution of the inputclassification image 320 (e.g., indicated by the transition betweenpixels 322 to pixels 332). In the illustrated case, the inputclassification image 320 is a single two-dimensional image of 24-by-25pixels, and the first set of output images 335 a includes thirty-twotwo-dimensional images of 12-by-13 pixels each.

Each level of the multi-level convolutional conversion 330 takes theoutput images 335 from the previous level as its input images, andcomputes its own output images 335 by applying a set of convolutionalfilters to its input images. Each level of the convolutional conversiongenerates a larger number of output images 335 than those computed inthe preceding level, and each of the output images 335 has a smallerresolution than the output images 335 computed in the preceding level.Each level can also apply the same or different convolutional filters togenerate its output images 335. The resulting output classificationimages can have a total of M output pixel values, where M is greaterthan N (i.e., M and N are positive integers, and M>N). The final levelof the multi-level convolutional conversion 330 can output some integernumber (D) of output classification images, each having a resolution ofE pixels by F pixels, such that D*E*F=M.

In the illustrated case, the second level of the multi-levelconvolutional conversion 330 computes a second set of output images 335b as a function of the first set of output images 335 a, so that thesecond set of output images 335 b includes sixty-four two-dimensionalimages of 6-by-7 pixels each. The third level of the multi-levelconvolutional conversion 330 computes a third set of output images 335 cas a function of the second set of output images 335 b, so that thethird set of output images 335 c includes 128 two-dimensional images of3-by-4 pixels each. Accordingly, the third set of output images 335 cgenerated by the third (final) level of the multi-level convolutionalconversion 330 includes 1,563 output pixel values (128*3*4=1,563).

Embodiments can serialize the M (e.g., 1,563) output pixel values to aone-dimensional (1-by-M) output classification array 345. Theserializing can be performed in such a manner that M element values ofthe output classification array 345 correspond to the M output pixelvalues of the final (e.g., third) set of output classification images335. As an example, each pixel of each output classification image 335is mapped, according to a predefined sequence, to a correspondingelement of the output classification array 345.

Embodiments can run the output classification array 345 through acomputational neural network 340 to compute a transition array 350. Thecomputational neural network 340 is trained to compute each of N elementvalues of the transition array 350 as a function of the M element valuesof the output classification array 345. Embodiments of the computationalneural network 340 are trained to compute an M-by-N set of weightings.According to such embodiments, computing the transition array 350includes applying the M-by-N weightings to the 1-by-M outputclassification array 345 to compute the 1-by-N transition array 350. Forexample, the output classification array 345 can be represented as a1-by-M matrix, the M-by-N weightings can be represented as an M-by-Nmatrix, and multiplying the 1-by-M matrix by the M-by-N matrix can yielda 1-by-N matrix corresponding to (e.g., equaling, or from which toderive) the 1-by-N transition array 350. For example, the computationalneural network 340 can be trained with a large number of sample sets ofimages corresponding to transition times at which programs transitionbetween not being part of the classified segment and being part of theclassified segment. In some implementations, the training data for thecomputational neural network 340 (ID training data 255 of FIG. 2) caninclude large numbers of classification model output 310 graph samples,along with feedback indicating appropriate transition times for each ofthose graph samples.

Embodiments can output a candidate transition time (represented as block355) as a function of the transition array 350. The candidate transitiontime 355 corresponds to a temporal location at which the mediaprogramming content transitions between not being part of the classifiedsegment and being part of the classified segment. In someimplementations, the computational neural network 340 is configured tooutput the transition array 350, such that each of the N element valuesof the transition array 350 corresponds to a respective one of the Nframe images (e.g., from the frame image output 215). In some cases, thecomputational neural network 340 includes further processing, such thatonly one of the N element values of the transition array 350 has a firsttransition element value (e.g., ‘1’) indicating a transition frame; andeach of the other (i.e., N−1) element values of the transition array hasa second transition element value (e.g., ‘0’) indicating anon-transition frame. In such implementations, outputting the candidatetransition time 355 as a function of the transition array 350 includesidentifying the one frame image of the N frame images as correspondingto the one of the N element values of the transition array 350 havingthe first transition element value. In some implementations, outputtingthe candidate transition time 355 further includes identifying atemporal location by mapping the identified one frame image to thetemporal location in the media programming content. For example, theidentified frame image can correspond to a particular time stamp that isa number of tenths of a second from the start of the program (or anyother suitable temporal measure).

In general, the data flow diagram 300 converts video data imageclassification results into a two-dimensional representation (an inputclassification image 320), runs the input classification image 320through a multi-level convolutional conversion 330 to output a set ofoutput images 355, runs a serialized representation of the output images355 (an output classification array 345) through a computational neuralnetwork 340 to generate a transition array 350, and derives a candidatetransition time 355 from the transition array 350. The specificimplementations of various aspects of the data flow diagram 300 shown inFIG. 3 are intended to be illustrative, and should not be considered aslimiting. Other implementations can modify the data flow diagram 300 andmay still achieve desirable results. For example, embodiments of themulti-level convolutional conversion 330 can include any suitable numberof levels (e.g., two, five, etc.), and each level can be configured tooutput any suitable number of output images 355 of any suitable numberand resolution. Similarly, embodiments can iterate in various ways. Oneimplementation converts the output classification array 345 into atwo-dimensional image (e.g., where the output of the multi-levelconvolutional conversion 330 includes 1,560 output pixel values, theoutput classification array 345 can be converted into a 39-by-40-pixelimage), and the two-dimensional image can be run back through themulti-level convolutional conversion 330 (or another multi-levelconvolutional conversion 330) to generate another output classificationarray 345. In some cases, selection of certain parameters (e.g., numberof levels of the multi-level convolutional conversion 330, iterations,etc.) can impact the amount of training (e.g., the amount ofclassification training data 235 used for training theauto-classification model 230 and/or the amount of identificationtraining data 255 used for training the auto-identification model 250)involved to achieve a particular level of accuracy in results.

Turning back to FIG. 2, embodiments can include a binge viewingcontroller 260 to generally control various aspects of interactionbetween video playback and a human user in a manner that supports abinge watching experience. Embodiments of the binge viewing controller260 can generate a playback navigation control, responsive to a userinterface command, to skip at least a portion of the classified segmentin accordance with the transition identification data 245 (e.g., thecandidate transition time). In one implementation, the media programmingcontent is a sequence of episodes including a first episode and a secondepisode following the first episode in the sequence, and the classifiedsegment is an end credits segment of the first episode (i.e., creditsrolling at the completion of the first episode). In such animplementation, the binge viewing controller 260 can generate theplayback navigation control to be displayed via a user interfaceresponsive to playback of the first episode reaching the candidatetransition time, such that, user interaction with the playbacknavigation control causes playback of the sequence of episodes to skipto the second episode past a remainder of the end credits segment of thefirst episode. For example, substantially at the same time that thecredits begin to roll, the playback interface can change to include aprompt, stating “Skip to next episode?”, or the like. Interacting withthe prompt (e.g., clicking an appropriate button on a remote control,touchscreen, etc.) can cause playback to skip the remainder of the endcredits and skip to the start of the next episode. The playbacknavigation control can be generated in any suitable manner, for example,at a pop-up, as an overlay, etc. The playback navigation control canalso include any suitable information, such as a countdown timer untilthe next episode will automatically begin.

Implementations having multiple types of segment classifications cansupport other functionality of the playback navigation control, and/orother permitted behaviors of the playback navigation control. As oneexample, the playback navigation control can be used in someimplementations to skip opening credits and/or recap scenes (e.g., clipsfrom previous episodes shown as a recap), etc. As another example, anembodiment may automatically classify segments of a program as includinga segment having the subject portion of the episode, followed by asegment of credits overlaid on a further subject portion of the episode,followed by a segment with credits rolling on a black background,followed by a segment having outtakes from the episode. In such a case,the playback navigation control may permit a user to skip one or moresegments (e.g., skip only the segment with credits rolling on a blackbackground), such that the user can watch the segment of creditsoverlaid on a further subject portion of the episode, and the segmenthaving outtakes from the episode. As another example, in an embodimentthat automatically classifies both ending and opening credits ofepisodes, the playback navigation control may permit a user to navigatefrom the end of a subject portion of a first episode directly to thebeginning of the subject portion of a second episode, skipping both theending credits of the first episode and the opening credits of thesecond episode.

A computer system as illustrated in FIG. 4 may be incorporated as partof the previously described computerized devices. FIG. 4 provides aschematic illustration of one embodiment of a computer system 400 thatcan implement various system components and/or perform various steps ofmethods provided by various embodiments described herein. It should benoted that FIG. 4 is meant only to provide a generalized illustration ofvarious components, any or all of which may be utilized as appropriate.FIG. 4, therefore, broadly illustrates how individual system elementsmay be implemented in a relatively separated or relatively moreintegrated manner.

The computer system 400 is shown comprising hardware elements that canbe electrically coupled via a bus 405 (or may otherwise be incommunication, as appropriate). The hardware elements may include one ormore processors 410, including, without limitation, one or moregeneral-purpose processors and/or one or more special-purpose processors(such as digital signal processing chips, graphics accelerationprocessors, video decoders, and/or the like); one or more input devices415, which can include, without limitation, a mouse, a keyboard, remotecontrol, and/or the like; and one or more output devices 420, which caninclude, without limitation, a display device, a printer, and/or thelike.

The computer system 400 may further include (and/or be in communicationwith) one or more non-transitory storage devices 425, which cancomprise, without limitation, local and/or network accessible storage,and/or can include, without limitation, a disk drive, a drive array, anoptical storage device, a solid-state storage device, such as a randomaccess memory (“RAM”), and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable and/or the like. Such storage devices maybe configured to implement any appropriate data stores, including,without limitation, various file systems, database structures, and/orthe like.

The computer system 400 might also include a communications subsystem430, which can include, without limitation, a modem, a network card(wireless or wired), an infrared communication device, a wirelesscommunication device, and/or a chipset (such as a Bluetooth™ device, an402.11 device, a WiFi device, a WiMax device, cellular communicationdevice, etc.), and/or the like. The communications subsystem 430 maypermit data to be exchanged with a network (such as the networkdescribed below, to name one example), other computer systems, and/orany other devices described herein. In many embodiments, the computersystem 400 will further comprise a working memory 435, which can includea RAM or ROM device, as described above.

The computer system 400 also can comprise software elements, shown ascurrently being located within the working memory 435, including anoperating system 440, device drivers, executable libraries, and/or othercode, such as one or more application programs 445, which may comprisecomputer programs provided by various embodiments, and/or may bedesigned to implement methods, and/or configure systems, provided byother embodiments, as described herein. Merely by way of example, one ormore procedures described with respect to the method(s) discussed abovemight be implemented as code and/or instructions executable by acomputer (and/or a processor within a computer); in an aspect, then,such code and/or instructions can be used to configure and/or adapt ageneral purpose computer (or other device) to perform one or moreoperations in accordance with the described methods.

A set of these instructions and/or codes might be stored on anon-transitory computer-readable storage medium, such as thenon-transitory storage device(s) 425 described above. In some cases, thestorage medium might be incorporated within a computer system, such ascomputer system 400. In other embodiments, the storage medium might beseparate from a computer system (e.g., a removable medium, such as acompact disc), and/or provided in an installation package, such that thestorage medium can be used to program, configure, and/or adapt a generalpurpose computer with the instructions/code stored thereon. Theseinstructions might take the form of executable code, which is executableby the computer system 400 and/or might take the form of source and/orinstallable code, which, upon compilation and/or installation on thecomputer system 400 (e.g., using any of a variety of generally availablecompilers, installation programs, compression/decompression utilities,etc.), then takes the form of executable code.

As an example, embodiments of the non-transitory computer-readablestorage medium include processor-readable instructions that cause one ormore processors 410 to perform various steps. The processor(s) 410 canextract N frame images at a sampling rate over an identified candidatesegment time window of the media programming content. The processor(s)410 can then associate each of the N frame images with a respectiveelement of a 1-by-N input classification array. Each element of theinput classification array can have an element value indicating a resultof automatically classifying the associated frame image into one ofmultiple predetermined classifications (e.g., including at least a firstclassification corresponding to the frame image being part of aclassified segment, and a second classification corresponding to theframe image not being part of the classified segment). The processor(s)410 can then convert the input classification array into atwo-dimensional input classification image having an input resolutioncorresponding to N, and a color depth corresponding to the predeterminedclassifications, such that N input pixel values of the inputclassification image correspond to the N element values of theclassification array. The processor(s) can then compute outputclassification images as a function of running the input classificationimage automatically through a multi-level convolutional conversion. Forexample, the output classification images can include a total of Moutput pixel values, where M is greater than N. The processor(s) 410 canthen serialize the M output pixel values to a 1-by-M outputclassification array, such that M element values of the outputclassification array correspond to the M output pixel values of theplurality of output classification images; and can compute a 1-by-Ntransition array as a function of running the output classificationarray through an computational neural network trained to compute each ofN element values of the transition array as a function of the M elementvalues of the output classification array. The processor(s) can thenoutput a candidate transition time as a function of the transitionarray, the transition time corresponding to a temporal location at whichthe media programming content transitions between not being part of theclassified segment and being part of the classified segment.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware might also be used, and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets, etc.), or both. Further, connection to othercomputing devices, such as network input/output devices, may beemployed.

As mentioned above, in one aspect, some embodiments may employ acomputer system (such as the computer system 400) to perform methods inaccordance with various embodiments of the invention. According to a setof embodiments, some or all of the procedures of such methods areperformed by the computer system 400 in response to processor 410executing one or more sequences of one or more instructions (which mightbe incorporated into the operating system 440 and/or other code, such asan application program 445) contained in the working memory 435. Suchinstructions may be read into the working memory 435 from anothercomputer-readable medium, such as one or more of the non-transitorystorage device(s) 425. Merely by way of example, execution of thesequences of instructions contained in the working memory 435 mightcause the processor(s) 410 to perform one or more procedures of themethods described herein.

The terms “machine-readable medium,” “computer-readable storage medium”and “computer-readable medium,” as used herein, refer to any medium thatparticipates in providing data that causes a machine to operate in aspecific fashion. These mediums may be non-transitory. In an embodimentimplemented using the computer system 400, various computer-readablemedia might be involved in providing instructions/code to processor(s)410 for execution and/or might be used to store and/or carry suchinstructions/code. In many implementations, a computer-readable mediumis a physical and/or tangible storage medium. Such a medium may take theform of a non-volatile media or volatile media. Non-volatile mediainclude, for example, optical and/or magnetic disks, such as thenon-transitory storage device(s) 425. Volatile media include, withoutlimitation, dynamic memory, such as the working memory 435.

Common forms of physical and/or tangible computer-readable mediainclude, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, any other physical medium with patterns of marks, a RAM, a PROM,EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any othermedium from which a computer can read instructions and/or code.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to the processor(s) 410for execution. Merely by way of example, the instructions may initiallybe carried on a magnetic disk and/or optical disc of a remote computer.A remote computer might load the instructions into its dynamic memoryand send the instructions as signals over a transmission medium to bereceived and/or executed by the computer system 400.

The communications subsystem 430 (and/or components thereof) generallywill receive signals, and the bus 405 then might carry the signals(and/or the data, instructions, etc., carried by the signals) to theworking memory 435, from which the processor(s) 410 retrieves andexecutes the instructions. The instructions received by the workingmemory 435 may optionally be stored on a non-transitory storage device425 either before or after execution by the processor(s) 410.

It should further be understood that the components of computer system400 can be distributed across a network. For example, some processingmay be performed in one location using a first processor while otherprocessing may be performed by another processor remote from the firstprocessor. Other components of computer system 400 may be similarlydistributed. As such, computer system 400 may be interpreted as adistributed computing system that performs processing in multiplelocations. In some instances, computer system 400 may be interpreted asa single computing device, such as a distinct laptop, desktop computer,or the like, depending on the context.

Systems including those described above can be used to implement variousmethods 500. FIG. 5 shows a flow diagram of an illustrative method 500for automated transition classification of media programming content,according to various embodiments. Embodiments of the method 500 begin atstage 504 by extracting N frame images at a sampling rate over anidentified candidate segment time window of the media programmingcontent. In some implementations, the identified candidate segment timewindow is J minutes long; and the sampling rate is K samples per second,such that N=J*K*60. As an example, the identified candidate segment timewindow is ten minutes long; and the sampling rate is one sample persecond, such that N is 600 frame images. In some implementations, thesampling rate is irregular. For example, the sampling rate may be set toextract 600 samples over a ten-minute window, but the time from onesample to the next may or may not by one second in every instance.

Some embodiments, at stage 508, include automatically classifying eachframe image into one of multiple by running each of the frame imagesthrough a machine learning model. The machine learning model can betrained as a function of a training set having a large number oftraining frame images associated with the classified segment, and alarge number of training frame images associated with a segment that isnot the classified segment. For example, a large number of credit andnon-credit frames can be used to train the machine learning model torecognize differences between credit and non-credit frames. In some suchimplementations, principle component analysis, or the like, is used by amachine to automatically identify which parameters are most useful indiscriminating between credit and non-credit frames (or any othersuitable classified segment of interest). In some implementations, themachine learning model can be trained to classify images according toonly one, or more than one, set of predetermined classificationscorresponding to only one, or more than one type of classified segment(e.g., ending credits, opening credits, etc.).

At stage 512, embodiments can associate each of the N frame imagesautomatically with a respective element of a 1-by-N input classificationarray. Each element of the input classification array can have anelement value indicating a result of automatically classifying theassociated frame image into one of multiple predeterminedclassifications (e.g., in stage 508). For example, the predeterminedclassifications include at least a first classification corresponding tothe frame image being part of a classified segment, and a secondclassification corresponding to the frame image not being part of theclassified segment. In some implementations, each element of the inputclassification array has either a first element value or a secondelement value (e.g., ‘1’ or ‘0’), the first element value indicatingthat the associated frame image is part of the classified segment, andthe second element value indicating that the associated frame image isnot part of the classified segment. For example, a ‘1’ can indicate thatthe frame image is classified as an credit frame (e.g., a frame that ispart of the end credits of a program), and a ‘0’ can indicate that theframe image is classified as a non-credit frame. In otherimplementations, more predetermined classifications can be used. Forexample, ‘0’ can indicate classification as a non-credit frame, ‘1’ canindicate classification as a credit frame with credits overlaid on asingle-color (e.g., black) background), and ‘2’ can indicateclassification as a credit frame with credits overlaid on otherprogramming in the background.

At stage 516, embodiments can convert the input classification arrayautomatically into a two-dimensional input classification image havingan input resolution corresponding to N and a color depth correspondingto the plurality of predetermined classifications. For example, N inputpixel values of the input classification image can correspond to the Nelement values of the classification array. In some implementations,converting the input classification array automatically into thetwo-dimensional input classification image at stage 516 includes mappingeach of the N element values of the classification array to a respectiveone of the N input pixel values of the input classification image, suchthat the input classification image has a resolution of B pixels by Cpixels, where B*C=N. As an example, the extracting at stage 504 canresult in 600 frame images, and the input classification image can be atwo-dimensional image of 24-by-25 pixels (24*25=600 pixels). In otherimplementations, each of the N elements of the input classificationarray can be mapped to more than one pixel of the input classificationimage. Each pixel can be colored according to its corresponding elementof the input classification array. For example, in implementationshaving a binary input classification array (i.e., having only twopossible values for each element, including a first value correspondingto one classification, and a second value corresponding to anotherclassification), the input classification image can be a two-color(e.g., black and white) image.

At stage 520, embodiments can compute output classification imagesautomatically as a function of running the input classification imagethrough a multi-level convolutional conversion. The resulting outputclassification images can have a total of M output pixel values, where Mis greater than N (i.e., M and N are positive integers, and M>N). Insome implementations, the multi-level convolutional conversion includesa number of levels of convolutional conversion. Each level takes theoutput images from the previous level as its input images, and computesits own output images by applying a set of convolutional filters to itsinput images. Each level of the convolutional conversion generates alarger number of output images than those computed in the precedinglevel, and each of the output images has a smaller resolution than theoutput images computed in the preceding level. Each level can also applythe same or different convolutional filters to generate its outputimages. The final level of the multi-level convolutional conversion canoutput some integer number (D) of output classification images, eachhaving a resolution of E pixels by F pixels, such that D*E*F=M. Forexample, after three levels of convolutional conversion, a24-by-25-pixel input classification image can be converted into 1283-by-4-pixel output classification images, such that there are128*3*4=1563 total output pixel values.

At stage 524, embodiments can serialize the M output pixel valuesautomatically to a 1-by-M output classification array. The serializingcan be performed in such a manner that M element values of the outputclassification array correspond to the M output pixel values of theplurality of output classification images. As an example, each pixel ofeach output classification image is mapped, according to a predefinedsequence, to a corresponding element of the output classification array.

At stage 528, embodiments can compute a 1-by-N transition arrayautomatically as a function of running the output classification arraythrough an computational neural network trained to compute each of Nelement values of the transition array as a function of the M elementvalues of the output classification array. Embodiments of thecomputational neural network are trained to compute an M-by-N set ofweightings. According to such embodiments, computing the transitionarray includes applying the M-by-N weightings to the 1-by-M outputclassification array to compute the 1-by-N transition array. Forexample, the output classification array can be represented as a 1-by-Mmatrix, the M-by-N weightings can be represented as an M-by-N matrix,and multiplying the 1-by-M matrix by the M-by-N matrix can yield a1-by-N matrix corresponding to (e.g., equaling, or from which to derive)the 1-by-N transition array.

At stage 532, embodiments can output a candidate transition time as afunction of the transition array. The candidate transition timecorresponds to a temporal location at which the media programmingcontent transitions between not being part of the classified segment andbeing part of the classified segment. In some implementations, thecomputational neural network is configured to output the transitionarray, such that each of the N element values of the transition arraycorresponds to a respective one of the N frame images; only one of the Nelement values of the transition array has a first transition elementvalue (e.g., ‘1’) indicating a transition frame; and each of the other(i.e., N−1) element values of the transition array have a secondtransition element value (e.g., ‘0’) indicating a non-transition frame.In such implementations, outputting the candidate transition time as afunction of the transition array includes identifying one frame image ofthe N frame images as corresponding to the one of the N element valuesof the transition array comprising the first transition element value.In some implementations, outputting the candidate transition time atstage 532 further includes identifying the temporal location by mappingthe identified one frame image to the temporal location in the mediaprogramming content. For example, the identified frame image cancorrespond to a particular time stamp that is a number of tenths of asecond from the start of the program (or any other suitable temporalmeasure).

The methods, systems, and devices discussed above are examples. Variousconfigurations may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods may be performed in an order different from that described,and/or various stages may be added, omitted, and/or combined. Also,features described with respect to certain configurations may becombined in various other configurations. Different aspects and elementsof the configurations may be combined in a similar manner. Also,technology evolves and, thus, many of the elements are examples and donot limit the scope of the disclosure or claims.

Specific details are given in the description to provide a thoroughunderstanding of example configurations (including implementations).However, configurations may be practiced without these specific details.For example, well-known circuits, processes, algorithms, structures, andtechniques have been shown without unnecessary detail in order to avoidobscuring the configurations. This description provides exampleconfigurations only, and does not limit the scope, applicability, orconfigurations of the claims. Rather, the preceding description of theconfigurations will provide those skilled in the art with an enablingdescription for implementing described techniques. Various changes maybe made in the function and arrangement of elements without departingfrom the spirit or scope of the disclosure.

Also, configurations may be described as a process which is depicted asa flow diagram or block diagram. Although each may describe theoperations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be rearranged. A process may have additional steps notincluded in the figure. Furthermore, examples of the methods may beimplemented by hardware, software, firmware, middleware, microcode,hardware description languages, or any combination thereof. Whenimplemented in software, firmware, middleware, or microcode, the programcode or code segments to perform the necessary tasks may be stored in anon-transitory computer-readable medium such as a storage medium.Processors may perform the described tasks.

Having described several example configurations, various modifications,alternative constructions, and equivalents may be used without departingfrom the spirit of the disclosure. For example, the above elements maybe components of a larger system, wherein other rules may takeprecedence over or otherwise modify the application of the invention.Also, a number of steps may be undertaken before, during, or after theabove elements are considered.

What is claimed is:
 1. A method for automated transition classificationof media programming content, the method comprising: generating an inputclassification image by assigning each of N pixels of the inputclassification image to a respective frame image of N frame imagesextracted from a portion of media programming content, and assigning toeach of the N pixels a respective pixel value indicating at leastwhether the respective frame image is classified as part of a classifiedsegment of the media programming content; first computing M elementvalues of an output classification array automatically as a function ofrunning the input classification image through a multi-levelconvolutional conversion; and second computing a transition arrayautomatically as a function of running the output classification arraythrough a computational neural network trained to compute each of Nelement values of the transition array as a function of the M elementvalues of the output classification array, such that the transitionarray indicates a temporal location at which the media programmingcontent transitions between not being part of the classified segment andbeing part of the classified segment.
 2. The method of claim 1, furthercomprising: outputting the temporal location.
 3. The method of claim 1,wherein: the transition array indicates a transition frame at which themedia programming content transitions between not being part of theclassified segment and being part of the classified segment, thetransition frame corresponding to one of the N image frames; and thesecond computing comprises mapping the transition frame to the temporallocation in the media programming content, the temporal locationcorresponding to a time stamp and/or a frame index in the mediaprogramming content.
 4. The method of claim 1, wherein: the assigning toeach of the N pixels the respective pixel value comprises computing therespective pixel value as a function of applying an auto-classificationmodel to the respective frame image, such that the respective pixelvalue corresponds to one of a plurality of predetermined classificationsdeterminable by the machine-implemented auto-classification model. 5.The method of claim 4, wherein: the machine-implementedauto-classification model is a machine learning model trained as afunction of classification training data comprising first trainingframes associated with the classified segment and second training framesnot associated with the classified segment.
 6. The method of claim 1,wherein: each respective pixel value indicates one of a number ofpredetermined classifications including whether the respective frameimage is classified as part of the classified segment; and generatingthe input classification image comprises generating a two-dimensionalimage having a resolution of at least N image pixels and a color depthcorresponding to at least the number of predetermined classifications.7. The method of claim 1, wherein the generating comprises: identifyinga candidate time window of the media programming content previouslypredicted to include a transition to or from the classified segment; andextracting the N frame images at a sampling rate over the identifiedcandidate segment time window.
 8. The method of claim 7, wherein thesampling rate is an irregular sampling rate.
 9. The method of claim 1,wherein the first computing comprises: running the input classificationimage through the multi-level convolutional conversion to compute aplurality of output classification images having a total of M outputpixel values, M being greater than N, each of the M element values ofthe output classification array corresponding to a respective one of theM output pixel values of the plurality of output classification images.10. The method of claim 9, wherein: the multi-level convolutionalconversion comprises a sequence of J convolutional conversions, each jthconvolutional conversion computing a jth plurality of output images as afunction of applying a jth plurality of convolutional filters to a jthset of input images; for each jth convolutional conversion, the jthplurality of output images being of a respective number larger than thatof the jth set of input images, and the jth plurality of output imageseach being of a respective resolution smaller than that of the jth setof input images; for j=1, the jth set of input images is the inputclassification image; and for each j greater than 1, the jth set ofinput images is the (j−1)th set of output images.
 11. The method ofclaim 10, wherein: for j=J, the jth set of output images is theplurality of output classification images computed, such that therespective number jth set of output images multiplied by the respectiveresolution of each of the jth set of output images equals M.
 12. Themethod of claim 1, wherein: the first computing comprises iterativelycomputing the M element values of the output classification array by: ina first iteration, running the input classification image through themulti-level convolutional conversion to generate the outputclassification array; and in each of at least one subsequent iteration,converting the output classification array to an output classificationarray image, and running the output classification array image throughthe multi-level convolutional conversion to re-generate the outputclassification array.
 13. The method of claim 1, wherein: the secondcomputing is such that one of the N element values of the transitionarray indicates a respective one of the N frame images as a transitionframe, and each of another N−1 of the N element values indicates anotherrespective one of the N frame images as a non-transition frame.
 14. Asystem for automated transition classification of media programmingcontent, the system comprising: one or more processors; and anon-transient, machine-readable medium coupled with the one or moreprocessors and having, stored thereon, instructions, which, whenexecuted, cause the one or more processors to perform steps comprising:generating an input classification image by assigning each of N pixelsof the input classification image to a respective frame image of N frameimages extracted from a portion of media programming content, andassigning to each of the N pixels a respective pixel value indicating atleast whether the respective frame image is classified as part of aclassified segment of the media programming content; first computing Melement values of an output classification array automatically as afunction of running the input classification image through a multi-levelconvolutional conversion; and second computing a transition arrayautomatically as a function of running the output classification arraythrough a computational neural network trained to compute each of Nelement values of the transition array as a function of the M elementvalues of the output classification array, such that the transitionarray indicates a temporal location at which the media programmingcontent transitions between not being part of the classified segment andbeing part of the classified segment.
 15. The system of claim 14,wherein the instructions, when executed, cause the one or moreprocessors to perform steps further comprising: outputting the temporallocation.
 16. The system of claim 14, wherein: the transition arrayindicates a transition frame at which the media programming contenttransitions between not being part of the classified segment and beingpart of the classified segment, the transition frame corresponding toone of the N image frames; and the second computing comprises mappingthe transition frame to the temporal location in the media programmingcontent, the temporal location corresponding to a time stamp and/or aframe index in the media programming content.
 17. The system of claim14, wherein: the assigning to each of the N pixels the respective pixelvalue comprises computing the respective pixel value as a function ofapplying an auto-classification model to the respective frame image,such that the respective pixel value corresponds to one of a pluralityof predetermined classifications determinable by the machine-implementedauto-classification model.
 18. The system of claim 14, wherein: eachrespective pixel value indicates one of a number of predeterminedclassifications including whether the respective frame image isclassified as part of the classified segment; and generating the inputclassification image comprises generating a two-dimensional image havinga resolution of at least N image pixels and a color depth correspondingto at least the number of predetermined classifications.
 19. The systemof claim 14, wherein the first computing comprises: running the inputclassification image through the multi-level convolutional conversion tocompute a plurality of output classification images having a total of Moutput pixel values, M being greater than N, each of the M elementvalues of the output classification array corresponding to a respectiveone of the M output pixel values of the plurality of outputclassification images.
 20. The system of claim 19, wherein: themulti-level convolutional conversion comprises a sequence of Jconvolutional conversions, each jth convolutional conversion computing ajth plurality of output images as a function of applying a jth pluralityof convolutional filters to a jth set of input images; for each jthconvolutional conversion, the jth plurality of output images being of arespective number larger than that of the jth set of input images, andthe jth plurality of output images each being of a respective resolutionsmaller than that of the jth set of input images; for j=1, the jth setof input images is the input classification image; and for each jgreater than 1, the jth set of input images is the (j−1)th set of outputimages.